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Abstract 

Background: Multisite qualitative studies are challenging in part because decisions regarding within-site and 
between-site sampling must be made to reduce the complexity of data collection, but these decisions may have 
serious implications for analyses. There is not yet consensus on how to account for within-site and between-site 
variations in qualitative perceptions of the organizational context of interventions. The purpose of this study was to 
analyze variation in perceptions among key informants in order to demonstrate the importance of broad sampling 
for identifying both within-site and between-site implementation themes. 

Methods: Case studies of four sites were compared to identify differences in how Department of Veterans Affairs 
(VA) medical centers implemented a Primary Care/Mental Health Integration (PC/MHI) intervention. Qualitative 
analyses focused on between-profession variation in reported referral and implementation processes within and 
between sites. 

Results: Key informants identified co-location, the consultation-liaison service, space, access, and referral processes 
as important topics. Within-site themes revealed the importance of coordination, communication, and collaboration 
for implementing PC/MHI. The between-site theme indicated that the preexisting structure of mental healthcare 
influenced how PC/MHI was implemented at each site and that collaboration among both leaders and providers 
was critical to overcoming structural barriers. 

Conclusions: Within- and between-site variation in perceptions among key informants within different professions 
revealed barriers and facilitators to the implementation not available from a single source. Examples provide insight 
into implementation barriers for PC/MHI. Multisite implementation studies may benefit from intentionally eliciting 
and analyzing variation within and between sites. Suggestions for implementation research design are presented. 



similarities in perspectives through either qualitative [5] or 
quantitative methods [6,7]. Some studies contrast the 
perspectives of key informants in analyses [8], but often 
the analysis of variation between informant types is not 
explicitly reported. Variation in perspectives among key 
informants may reflect substantive variation in how an 
intervention is implemented both within sites and be- 
tween sites. Because perceptions can vary among key 
informants, understanding the nature and extent of this 
variation may be critical in determining why interventions 
succeed or fail. The purpose of this study was to demon- 
strate the importance of broad sampling in implementa- 
tion studies by highlighting the role of qualitative 
variation in making inferences regarding both within-site 
and between-site implementation themes. 

Most qualitative implementation research has one of 
three foci. First, some qualitative studies are conducted 



Background 

Qualitative research methods explicitly recognize that 
knowledge is incomplete, or 'situated,' and that analysis of 
multiple diverse perspectives across informants can en- 
hance understanding of complex phenomena [1]. Studies 
of intervention implementation and organizational con- 
text vary widely in the analytical strategies used to account 
for this variation in perspectives. Some studies of imple- 
mentation and organizational context incorporate 
multiple informants [2,3], whereas others rely on single- 
informant designs [4]. Some studies divorce respondents 
from their organizational context in order to summarize 
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within the context of a randomized controlled trial 
[9,10]. Second, some qualitative studies investigate the 
factors that influence the spread of evidence-based prac- 
tice across facilities or regions [11-13]. Third, some 
qualitative studies are designed to understand the factors 
that influence local adoption of evidence-based practices 
[6,14,15] and local tailoring of interventions [16]. 
Gathering rich data regarding how the organizational 
context affects the implementation process is important 
in all three areas [15,17]. Organizational context can 
affect factors such as implementation fidelity, processes 
of implementation, and intervention effectiveness. Quali- 
tative research can be used to understand these factors 
in randomized controlled trials [18], studies of inter- 
vention spread [19], and adoption of evidence-based 
practices [20,21]. Research on the role of organizational 
context is particularly important in multisite implemen- 
tation studies as local organizational conditions present 
different opportunities and challenges at each site. 

Although the implementation science literature recognizes 
that organizational context impacts intervention imple- 
mentation, there is not yet consensus on how to account 
qualitatively for within-site and between-site variation in 
organizational context. Many multisite implementation 
studies do not include qualitative methods [22-26], and 
those few that do tend to sample broadly across sites 
(and narrowly within a site) in order to understand 
between-site variation [27-29]. However, characteristics 
of informers, such as profession, leadership level, and 
role in the intervention, may influence perceptions of 
an intervention at a single site that may or may not 
generalize across multiple sites within the same study 
[30]. This variation is challenging for qualitative imple- 
mentation research because it increases the cost and 
complexity of data analysis. To reduce this cost and 
complexity, many multisite qualitative implementation 
researchers make decisions that restrict sampling (e.g., de- 
creasing the length of engagement with a site or limiting 
sampling across different types of key informants) in order 
to manage the time and expense associated with multisite 
qualitative research. Restricted sampling in multisite quali- 
tative implementation studies may be better than no 
multisite research at all, but it is important to carefully 
consider the tradeoffs researchers make between analytic 
and sampling strategies. 

To facilitate the qualitative study of intervention imple- 
mentation across sites, the authors developed a framework 
for analyzing within-site and between-site variation based 
on the literature on qualitative triangulation. The current 
study will apply this framework by analyzing the variation 
in how an intervention implementation was perceived by 
different sets of key informants within multiple sites. This 
may be particularly important in cases where the relevant 
implementation crosses professional boundaries, so we 



chose a multidisciplinary intervention that affects both 
primary care and mental health staff at a set of medical 
facilities. Results may be helpful to both qualitative 
researchers involved in multisite studies and intervention 
researchers who are required by funding agencies to 
include a qualitative evaluation component in multisite 
implementation studies. 

Methods 

Intervention 

Primary Care/Mental Health Integration (PC/MHI) is a 
national effort in the Department of Veterans Affairs 
(VA) to improve the management of mental health in 
VA outpatient settings. PC/MHI was initiated in 2008 by 
a national mandate. Most PC/MHI programs (93%) 
include co-located mental health providers to increase 
the availability of mental health services in the primary 
care setting [31]. 

Participant and clinic characteristics 

We analyzed archival data drawn from a 2009 manage- 
ment evaluation that was designed to understand the 
processes used to provide co-located PC/MHI in a sample 
of hospital-based and affiliated freestanding outpatient 
clinics [32] . The VA Institutional Review Board approved 
these analyses. From the larger evaluation data set, we 
purposefully selected four clinics. Seventeen employees 
were interviewed in the selected clinics as part of the 
larger evaluation study. Clinic leaders and staff from 
both primary care and mental health were interviewed 
at each site. The four sampled sites were split between 
two different VA medical centers; each medical center 
is a quasi-independent facility with its own budget and 
management structure. Within each medical center, we 
selected one hospital-based primary care clinic and one 
large (10,000+ unique patients) community-based out- 
patient clinic (CBOC). The national PC/MHI mandate 
has similar standards for these large clinics. The four 
clinics in the current study were selected because the 
interviews demonstrated both within-site and between-site 
variation and because the affiliation between the 
hospital-based and freestanding outpatient clinics 
facilitates analysis of the variability within a single 
organizational authority structure. Thus, results will 
also show whether implementation factors varied across 
administratively connected sites. To protect confidenti- 
ality, sites were labeled for the current study as 
"Alpha," "Bravo," Yankee," and "Zulu," where Site Alpha 
was a hospital-based site affiliated with the same med- 
ical center as CBOC Bravo, and hospital-based Yankee 
was affiliated with CBOC Zulu. Key informants at each 
site included the specialty mental health leader (MHLs), 
staff providing PC/MHI care (PCMH), and primary 
care physicians (PCPs). 
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Data collection 

Semi-structured interviews were used to conduct a forma- 
tive evaluation between July and August 2009. Formative 
evaluation was defined as an assessment of both potential 
and actual influences on implementation progress and 
effectiveness [33]. The purpose of the interviews was to 
understand how referral processes were implemented to 
coordinate mental healthcare between primary care and 
mental health staff. Stratified purposeful sampling was 
used to identify key informants [34] . We interviewed clinic 
leaders, who then identified staff who were the most 
knowledgeable regarding the PC/MHI implementation. 

Interviews were collected from MHLs, PCMH, and 
PCPs. Telephone interviews were conducted for up to 
45 minutes, with a note taker instructed to record 
responses verbatim where possible. To protect informant 
confidentiality, voice recording was not conducted. Data 
were collected until between-site saturation was reached. 
That is, saturation was judged to occur when interviews 
reported implementation issues that paralleled issues at 
other sites and no new implementation issues were 
reported. At most sites, interviews were conducted with 
each of two PCMHs and two PCPs. To protect confiden- 
tiality, and because the divergence within informant type 
(e.g., between two physicians at one site) was not as 
striking as the divergence between informant type (e.g., 
between PCMH and PCP at one site), within-type diver- 
gence will not be presented. In other words, we will 
present results from MHLs, PCMH, and PCPs, but not 
discuss variation within PCMH and within PCP responses. 
Instead, we focus on variation between positions and be- 
tween sites. We will present results as if the PCMH and 
PC perceptions were each provided by one informant. 

Using a case study model [35], an interview protocol 
was developed for the evaluation study that included 
open-ended questions that allowed the respondents to 
describe processes of care used in that site and to describe 
how those processes changed over time [36] with follow- 
up questions [37,38]. The evaluation team developed 

Table 1 Interview questions and specific probes 

Interview question 



seven interview questions and specific probes to gather in- 
formation on the processes used to coordinate primary 
care and mental health staff (see Table 1). Interviewers 
first asked two "grand tour" questions [39], which allowed 
informants to describe the present and evolved/past 
processes of care used in their sites openly and without a 
researcher-imposed framework. Interviews then turned 
toward interviewer-driven topics related to coordination, 
which was a focal point for the intervention and its 
evaluation. 

Triangulation of perceptual variation 

Study analyses required researcher decisions regarding 
how to aggregate perceptions that were shared or 
unique. The purpose of these analyses was to identify 
common themes regarding the implementation (e.g., 
leaders are important facilitators), while coding variation 
within these themes. Perceptions may therefore either 
converge toward these common themes or diverge 
toward a unique perspective [40]. 

Generally, the credibility of qualitative research can be 
established through prolonged engagement, persistent ob- 
servation, and triangulation. [1] a Prolonged engagement 
(i.e., research over an extended time period) provides a 
broad scope of understanding. Persistent observation (i.e., 
focused investigation on specific topics) provides depth to 
the understanding of a specific phenomenon (Lincoln & 
Guba, p. 301-307). Triangulation involves a comparison 
of different kinds of evidence to evaluate the credibility of 
the evidence. 

Triangulation used to support convergence across sources 
may be conceptualized as a validity test where perceptions 
that are not corroborated are seen as questionable. Conver- 
gence is often valued in implementation research because 
parallel perspectives across key informants can increase the 
validity of the data [41], but divergent perspectives are not 
necessarily invalid [42]. Triangulation may also be used to 
support divergence across sources by identifying patterns of 
distortion [43] common to particular sources and using those 



Specific probes 



1. Imagine that a patient with depression symptoms comes to the clinic. Can you walk Referral process, differences between diagnoses 
me through a typical process of care? 

2. How has this process changed over the past 10 years (or since you arrived in the Recent changes, leadership support, referrals, interpersonal 
clinic)? interactions, physical structure 

3. Tell me about your sense of the need for coordination between primary care and Examples of good and poor coordination 
mental health. 



4. How would you change your clinic to better coordinate care? 



Communication, collaboration, resource barriers 



5. Have you or anyone you know had to develop your own coordination procedures to Workarounds, ad-hoc coordination procedures 
ensure that patients receive the best care? 

6. Can you tell me about the relationship between the people in the primary care and Face-to-face contact, trust 
mental health clinics? 



7. In what situations would you say that teamwork is most important? 



Coworkers back each other up 
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patterns to evaluate the credibility of the divergent 
perceptions. Few implementation studies that sample 
multiple key informants discuss the implications of diver- 
gence for intervention implementation, but differences 
between informant groups may be as important as similar- 
ities in understanding why an intervention succeeded, 
failed, or needed to be tailored for a specific site. Diver- 
gent perceptions indicate that an informant or group 
of informants interprets a shared phenomenon differ- 
ently and can suggest the need for additional analyses, 
data collection, or follow-up clarifications to under- 
stand the meaning of the divergence. Failing to identify 
this variation in implementation studies may result in 
an incomplete understanding of the factors impacting 
intervention implementation. Convergent perceptions 
support understanding the multidimensional nature of 
implementation, whereas divergence can support rejec- 
tion of a hypothesis, facilitate hypothesis construction, 
or highlight differences between sources [40]. 

Analysis of within-site and between-site variation 

The analytic strategy for the current study was developed 
following initial data analysis (for evaluation purposes) in 
which discrepancies across informants emerged. During 
that initial round of analyses, two coders sequentially 
analyzed the evaluation data, evaluated conceptual 
codes, revised codes, and resolved discrepancies [44]. 
During this analysis, we discovered substantial within- 
site and between-site variation in informants' accounts 
of (1) the specific details of how co-located care was 
implemented and (2) the referral processes used to co- 
ordinate between primary care and mental health. The 
analyses for this study focused on exploring variation 
across informants in those two areas. Based on the ini- 
tial data analysis, the authors collaboratively selected 
two medical centers to analyze more deeply {i.e., the 
four sites selected for the current study). These sites 
demonstrated particularly clear examples of variation 
within and between sites. Data were then reanalyzed for 
the purpose of identifying both within-site and between- 
site implementation themes. 

For the purposes of demonstrating how these analyses 
were conducted, we make a distinction between redun- 
dant, convergent, and divergent perceptions. Redundant 
perceptions provided information that was completely 
shared across informants (we note that complete agree- 
ment is a strict criterion adopted for the purpose of 
demonstrating what was truly shared across informants). 
Convergent perceptions provided information that 
elaborated redundant perceptions by demonstrating vari- 
ation in interpretation of shared phenomena. For 
example, primary care and mental health may perceive 
different aspects of the same phenomenon. Finally, di- 
vergent perceptions reflected a unique departure from 



the convergent perceptions. Divergence suggests that a 
phenomenon is interpreted differently across sources. 

Within each site, we analyzed the data from the 
perspectives of the three key informant groups to identify 
patterns of variation that distinguished one source from an- 
other [45]. One qualitative researcher categorized data as 
redundant, convergent, or divergent. Additional researchers 
then reviewed the transcripts and, through discussion, 
came to a consensus on the interpretation of the data. 
The researchers evaluated the credibility of divergent 
perceptions by analyzing both the pattern of responses 
within sites (i.e., does the divergence fit with the responses 
of other informants) and also the patterns across sites 
(e.g., does the divergence fit the types of responses from 
similar informants across sites). Within-site implemen- 
tation themes were inferred from the convergence and 
divergence of responses among the key informant 
groups at a site. Between-site implementation themes 
were inferred from the patterns of responses across the 
four sites rather than simply comparing the different 
within-site themes. The analysis of convergent and di- 
vergent data used to identify implementation themes 
are described below. 

Results 

Summary data are presented in Table 2 for Site Alpha, 
Table 3 for Site Bravo, and Table 4 for Site Yankee. 
Informants at Site Zulu provided unique, but not neces- 
sarily divergent, perceptions and are discussed separ- 
ately. Tables present the redundant concepts at each site 
(i.e., the descriptive data that were supported by 
perceptions from all three informants) and the convergent 
and divergent perceptions that elaborated the redundant 
concepts. The analyses below reflect the presentation in 
the tables. Redundant perceptions are described first, 
followed by the convergent and divergent perceptions. 
After the within-site themes are described at each site, we 
present an analysis of the between-site implementation 
theme. 

Across the sites, we found that MHL reports generally 
agreed with PCMH and PCP reports, providing some 
support for the use of single-source MHL interview 
methodologies. The redundant concepts were particu- 
larly salient characteristics of PC/MHI implementation, 
but the aspects of these characteristics that were agreed 
upon were often of trivial importance to the implemen- 
tation. Instead, the convergent and divergent perceptions 
of these redundant characteristics were critical to identify- 
ing implementation themes. PCMH informants provided 
the most detailed reports of the procedures, barriers, and 
facilitators for the implementation. This result was 
expected since they were the staff most directly tasked 
with implementation of the process. The perceptions 
provided by PCPs appeared to offer the most unique 
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Table 2 Convergence and divergence in referral procedures for Site Alpha 



Redundant concept 


Convergence or 
divergence 


Supporting data 


Psychologists are co-located 


Convergence 


MHL: Multiple revisions to service agreements were needed to tailor the implementation to 


in primary care 




primary care needs. 




Divergence 


PCP: PC/MHI could be expanded to help manage difficult medical patients. 


Traditional consultation- 


Convergence 


MHL: PC/MHI supplements the consultation-liaison (POD) services. 


liaison service 


Convergence 


PCMH: POD services are separate from the implementation. 




Divergence 


PCMH: POD restricts access to specialty mental health. 




Divergence 


PCP: POD availability has improved since the implementation, but POD sometimes refers patients 






back to PC without treatment. 


mplementation limited by a 


Convergence 


PCMH: Reported a growth in referrals since implementation and implemented innovative new 


lack of space 




procedures to make use of limited co-located staffing. 




Convergence 


PCP: Acknowledged improved access, but doubted long-term impact due to lack of space. 



Note: Supporting data are presented as either convergent or divergent with the redundant concept. MHL refers to the mental health leader at the site; PCMH 
refers to the Primary Care/Mental Health Integration (PC/MHI) informants at the site; PCP refers to the primary care physician informants at the site. 
POD = psychiatrist on duty. 



information, often capturing important contextual details 
that were not provided by other sources. Together, the 
perceptions gathered from these three sources {i.e., MHL, 
PCMH, and PCP) facilitated site-level inferences about 
the implementation of PC/MHI that would not be avail- 
able from any single source. 

Site Alpha 

Three redundant PC/MHI concepts were identified at 
Site Alpha (Table 2). First, all informants agreed that 
the co-location of psychologists in the primary care set- 
ting was a key component of the PC/MHI intervention. 
Second, all informants acknowledged that the trad- 
itional consultation-liaison service was available; co- 
located psychologists were an additional mechanism for 
mental health access. Third, all informants reported 
that the PC/MHI intervention was limited by a lack of 
space. 



Psychologists are co-located 

The MHL informant elaborated upon the common ob- 
servation that psychologists are co-located by noting 
how the intervention required multiple revisions to the 
service agreements (i.e., formal policies that standardize 
responsibilities between services) to fit with the needs of 
primary care. This convergent perception suggests the 
potential difficulties involved in adapting an interven- 
tion to existing organizational procedures. PCPs also 
acknowledged the co-located structure of PC/MHI, but 
generally expressed frustration with the intervention. 
PCPs noted that the goal of PC/MHI should be helping 
PCPs care for patients. PCPs suggested that PC/MHI 
would benefit from expanding to helping manage diffi- 
cult medical patients, rather than focusing on specific 
mental health conditions. This divergent perspective 
indicates that the PCP and MHL representatives did 
not share the same opinion regarding the role of the 



Table 3 Convergence and divergence in referral procedures for Site Bravo 



Redundant concept 


Convergence or 
divergence 


Supporting data 


PC/MHI has improved access 


Convergence 


MHL: Co-location and the size of the clinic promote positive interactions, as providers see 
each other at lunch and at meetings. 




Convergence 


MHL: PC/MHI goal is immediate access; PCMH is always available. 




Convergence 


PCMH: Conducting a pilot study to provide access to walk-in patients. 




Divergence 


PCMH: Getting buy-in from PC is biggest challenge; informal discussions in the lunch room 
and "selling ourselves" increased curbside consults. 




Divergence 


PCP: Psychiatrists resisted helping PC manage behavioral aspects of chronic diseases, but 
negotiations have resulted in progress. 


Referrals include standard referrals 
and curbside consults 


Divergence 


PCMH: Nurse routine screening often initiates referrals and some nurses refer patients 
inappropriately; working with nurse manager to educate staff. 




Convergence 


PCP: Norms indicate that knocking on doors is appropriate, even if it interrupts ongoing 
psychological care. 



Note: Supporting data are presented as either convergent or divergent with the redundant concept. MHL refers to the mental health leader at the site; PCMH 
refers to the Primary Care/Mental Health Integration (PC/MHI) informants at the site; PCP refers to the primary care physician informants at the site. 
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Table 4 Convergence and divergence in referral procedures for Site Yankee 



Redundant concept 



Convergence or Supporting data 
divergence 



Mental health providers are co-located in 
the primary care setting 



Convergence MHL: Co-location promotes communication and shared understanding, but lack of 
space limited the implementation. 

Convergence PCMH: Only three primary care teams have co-located PCMH due to space 
restrictions. 

Divergence PCP: Some primary care providers forced to give up space for PC/MHI — created 

conflicts; implementation planning was conducted unilaterally by MR 

Convergence PCMH: PC referrals to PCMH have increased, particularly for medical conditions 
such as diabetes management. 

Convergence PCP: Implementation increased patient compliance with referrals. 

Divergence PCP: PCMH helps refer appropriately, but some providers may not refer to PCMH at 

all. 

Divergence PCP: Conflict between PCMH and ER regarding responsibility for urgent patients. 

Note: Supporting data are presented as either convergent or divergent with the redundant concept. MHL refers to the mental health leader at the site; PCMH 
refers to the Primary Care/Mental Health Integration (PC/MHI) informants at the site; PCP refers to the primary care physician informants at the site. ER = 
emergency room. 



PC/MHI team leader triages consults and 
assigns them to a PC/MHI provider 



co-located psychologist. That is, there was variation in 
whether informants felt that PC/MHI should be 
targeted specifically to mental health conditions or 
more broadly targeted. A PCP suggested that "old- 
fashioned team meetings" were needed to provide col- 
laborative care for patients. This divergent perception 
reflects a general theme that cross-service collaboration 
was not part of the PC/MHI implementation at Site 
Alpha. 

Consultation-liaison service 

The traditional psychiatrist on duty (POD) consultation- 
liaison service was mentioned by the three key informants, 
but perceptions of POD interactions varied greatly. 
MHL convergent perceptions suggested that PC/MHI 
supplemented the traditional consultation-liaison ser- 
vice by providing quick access for common mental 
health needs. PCMH agreed with MHL that the trad- 
itional POD consultation-liaison service was important 
for mental health access, but noted that PC/MHI 
services were not integrated with POD; rather, PC/MHI 
was a separate intervention added to the preexisting 
POD services. PCMH elaborated with a divergent per- 
spective that the POD has a gatekeeper role and 
restricts, rather than facilitates, mental health access. 
PCP supported this perception and indicated that the 
POD referred patients back to PCP without treatment. 
This divergent perception suggests that the POD and 
PC/MHI were viewed as separate and potentially 
conflicting methods for facilitating mental healthcare in 
the primary care setting. 

Lack of space 

Space was identified as a barrier by all three informants, 
but the effects varied. MHL reported space limited the 
number of co-located psychologists. PCP reported space 



created barriers to collaborative care and doubted the 
long-term impact of PC/MHI without collaboration. 
PCMH elaborated on the effect of the space limitation, 
reporting a growth in referrals since the implementation 
that created a screening bottleneck. Because no additional 
co-located staff was possible due to limited space, PCMH 
responded by developing innovative solutions to increase 
access with a minimal number of co-located psychologists. 
Specifically, PCMH tested a group screening procedure 
that decreased the time between referrals from PCP to 
PCMH to follow-up care (e.g., POD, substance abuse, 
posttraumatic stress disorder referral). Thus, convergent 
perceptions suggested that the effectiveness of PC/MHI at 
Site Alpha may partially depend on local tailoring of 
procedures to overcome space barriers. 

Dominant implementation theme at Site Alpha 

Variation in perceptions suggested a within-site theme 
that processes of coordination between services are crit- 
ical to integrated care. To implement co-located care, 
PCMH reported engaging primary care, and PCP 
reported overcoming psychiatrist resistance. Regarding 
the consultation-liaison service, MHL indicated that the 
POD provided an alternative mechanism to access men- 
tal health services. Divergent perceptions elaborated that 
the POD was not well integrated with PC/MHI, and the 
POD process served a gatekeeper role conflicting with 
PC/MHI. The credibility of this conclusion was 
enhanced by both PCMH and PCP reporting that the 
preexisting role of psychiatrists/POD was a barrier to PC/ 
MHL Regarding lack of space, convergent perceptions 
from PCMH elaborated how innovative processes for 
mental health screening were developed on the frontline 
to handle high demand to overcome lack of space. Finally, 
there was divergence over the extent to which PC/MHI 
planning was one-sided, raising important questions about 



Benzer et al. Implementation Science 2013, 8:4 
http://www.implementationscience.eom/content/8/1/4 



Page 7 of 14 



which types of patient care should be shared between 
primary care and mental health (i.e., should PC/MHI 
address medical comorbidities?). Results suggested that 
the coordination processes, rather than structural factors 
(i.e., co-location and space), facilitate and/or restrict PC/ 
MHI implementation at Site Alpha. 

Site Bravo 

Site Bravo (Table 3) is a community-based clinic that 
shares a common leadership structure with Site Alpha. 
Two redundant concepts were identified at Site Bravo. 
First, all informants agreed that the co-location of 
psychologists in the primary care setting improved 
access to mental health care. Before the implementation 
of PC/MHI, patients needing mental health treatment 
would travel more than an hour to Site Alpha for an ap- 
pointment. Second, all informants reported that the PC/ 
MHI intervention improved same-day mental health 
access though the processes of traditional referrals and 
"curbside consults." 

PC/MHI improves access 

MHL agreed that co-locating mental health providers 
next to primary care improved access. Convergent 
perceptions from MHL indicated that co-location and 
the size of the clinic promoted positive interactions, 
such as shared lunches and formal meetings. PCMH 
provided a divergent perspective for positive interactions 
and noted that getting "buy-in" from primary care 
providers was the biggest challenge. Contrary to the 
MHL perception that co-location and size had passive 
effects on referrals and curbside consults, PCMH 
actively used opportunities to "sell" themselves (e.g., 
through informal discussions in the lunchroom). PCMH 
agreed that the goal of PC/MHI was immediate access 
and reported conducting a pilot study to provide access 
for walk-in patients. PCP provided a second divergent 
perspective, describing psychiatrists working in specialty 
mental health as one source of the resistance to PC/MHI 
implementation. According to PCP, the psychiatrists 
initially resisted helping PCPs manage the behavioral 
aspects of chronic diseases, but negotiations resulted in 
changes. Thus, several factors influenced access to PC/ 
MHI care at Site Bravo, but perceptions across the 
three informants suggested that the implementation 
was dependent on inter-service collaboration. 

Same-day access 

MHL provided a redundant perspective that the goal of 
PC/MHI was to provide immediate mental health access 
and that PCPs could choose to treat mental health 
conditions themselves or make formal referrals or informal 
"curbside consults." PCP provided convergent perceptions 
that elaborated on the meaning of informal curbside 



consults (e.g., knocking on a psychologist's door for an 
informal consult that complemented formal consults). 
PCMH provided convergent perceptions that further 
elaborated same-day access to PC/MHI. PCMH reported 
that referrals often were initiated with nurse screenings; 
however, some nurses refer patients inappropriately. 
PCMH reported working with the nurse manager to edu- 
cate staff to improve and create an appropriate referral 
processes. This convergent perception indicated that 
training may be a missing component of the intervention 
at some sites. 

Dominant implementation theme at Site Bravo 

Variation in perceptions suggested a within-site theme 
that collaboration between services facilitated the imple- 
mentation. MHL suggested that the structural aspect of 
the intervention (i.e., co-location of mental health 
providers) facilitated access, whereas PCMH and PCP 
noted the importance of process factors (e.g., active 
selling, psychiatrist resistance). In contrast to Site Alpha, 
these differences were attenuated by evidence of collab- 
oration across services. PCP indicated that MHL was 
aware of their concerns, and primary care was either 
informed or involved with mental health planning (i.e., 
negotiations). Mutual awareness suggests good commu- 
nication regarding PC/MHI across the services. Collab- 
oration was further evidenced by the interactions 
between PCP and MHL. Knocking on doors was an 
accepted method of receiving informal consults, and 
PCMH was comfortable providing feedback and training 
to PCP nurses regarding appropriate screening and 
referrals. Results suggest that collaboration across 
services facilitated implementation. 

Site Yankee 

Two redundant concepts were identified at Site Yankee 
(Table 4). First, all informants agreed that the co- 
location of psychologists in the primary care setting was 
a key component of the PC/MHI intervention. Ten years 
prior to the interviews, Site Yankee had a co-located 
psychologist, but the growth of primary care created 
demands for space that resulted in the psychologist 
moving with the rest of mental health to a building one 
mile away. The integrated care mandate resulted in four 
rooms being allocated for two co-located psychologists 
and two psychiatrists to provide same-day consults to 
primary care patients for evaluation and short-term 
interventions. Second, all informants acknowledged that 
referrals were conducted through a triage system where 
the PC/MHI team leader assigned consults. 

Lack of space 

All informants reported that space was a limitation, with 
PCMH indicating that only three primary care teams 
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within the site had co-located mental health providers. 
In contrast to MHL and PCMH, who each indicated 
space was a structural barrier that limited full staffing 
for the intervention, PCP reported a severe conflict be- 
tween primary care and mental health because some 
PCPs were forced to give up their space. PCP attributed 
the problem to MHL unilaterally planning for imple- 
mentation. This divergence between informants suggests 
a poor collaboration between primary care and mental 
health that may pose a serious threat to implementation 
sustainability. 

Triage referral system 

All informants reported that Site Yankee used a triage 
system for assigning patients to mental health providers. 
PCMH elaborated the referral process, suggesting that 
an increase in referrals in recent months, particularly for 
medical conditions such as diabetes, was a sign of imple- 
mentation success. PCP complemented the PCMH view 
that the implementation was successful, but suggested 
that the reason for this success was increased patient 
follow-up on mental health referrals. Further, PCP 
suggested intervention spread was a concern at Site 
Alpha. PCP reported that PC/MHI providers were valu- 
able in helping PCPs navigate the extensive mental 
health treatment options at the site, though some PCPs 
may not have referred to PC/MHI at all. PCP also 
reported divergent perceptions regarding a conflict be- 
tween the PC/MHI providers and the emergency room 
(ER). According to PCP, ER perceived that mental health 
urgent care should be the responsibility of PC/MHI, 
whereas PC/MHI felt that these patients should still be 
treated in ER. This conflict was resolved with assistance 
from leadership after a period of referring patients back 
and forth to PC/MHI and the ER. The ER example 
indicates that the PC/MHI intervention may conflict 
with existing procedures, and leaders should be aware of 
this potential and be ready to resolve conflicts. 

Dominant implementation theme at Site Yankee 

Variation in perceptions suggests a within-site theme 
that poor collaboration between primary care and men- 
tal health leaders has limited the implementation. In par- 
ticular, PCP divergent reports regarding the conflict over 
space suggests a lack of collaborative planning for the PC/ 
MHI implementation, which may have slowed interven- 
tion spread through primary care. Similar divergences 
between PCP and MHL regarding a lack of collabora- 
tive planning were evidenced at Site Alpha, suggesting 
a pattern of divergence between primary care and men- 
tal health across sites. Conflict over space is a particular 
concern for intervention sustainability at Site Yankee, 
given that a co-located care program was ended 10 
years previously due to a lack of space. A lack of 



collaboration between primary care and mental health 
was further evidenced by differences in the perceived 
success of PC/MHI. The conclusion that Site Yankee 
implementation was successful implementation is 
supported by the convergence between PCMH and 
PCP, but interviewing multiple informants provided a 
richer understanding of the boundaries of this success. 
For example, the evidence for success from the PCP 
perspective might be characterized as patient-centered 
(e.g., increased patient follow-up), whereas evidence 
from the PCMH perspective might be characterized as 
provider-centered (e.g., increased referrals). These results 
support a broad within-site implementation theme of poor 
collaboration between primary care and mental health. 

Site Zulu 

Site Zulu is a community-based clinic that shares a com- 
mon leadership structure with Site Yankee. Primary care 
and mental health had previously been located in the same 
building, but increased staffing related to the implementa- 
tion of PC/MHI created space restrictions that were 
solved by moving all non-medicating PC/MHI providers 
to a separate building over a mile away. Psychiatrists are 
still co-located in primary care to provide medication sup- 
port to primary care physicians. PC/MHI implementation 
is based on electronic referrals. Referrals are sent to the 
mental health nurse practitioner, who triages and assigns 
them to a mental health provider. Communication back 
to primary care is done through additional signers on the 
electronic medical record. Informants agreed that the im- 
plementation did not change the referral procedures. 
Rather, the key aspect of the intervention was reported to 
be the increased staffing that speeds the referral process. 

Unique perceptions 

Site Zulu was unique among the sampled sites due to 
geographical separation between primary care and the 
non-medicating providers. Redundant perceptions were 
common among informants. A limited amount of con- 
vergent information elaborating referral processes was 
collected at this site, but the information lacked a com- 
mon theme. MHL stated that local VA regulations had 
previously required mental health providers to approve 
psychiatric medications, but procedures had been 
changed to allow primary care physicians more responsi- 
bility. PCMH was the only informant to note that 
patients can be discharged from the hospital with PC/ 
MHI appointments, a valuable process for post-discharge 
continuity of care. PCP revealed a coordination problem 
involving psychiatric admissions. Local police would not 
escort patients to the psychiatric hospital because it is 
located in a separate county, and procedures were 
developed where a psychiatric nurse would escort patients 
to the hospital. 
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Dominant implementation theme at Site Zulu 

Site Zulu presents a unique model of PC/MHI compared 
to the other sites. Co-location was limited to psychiatrist 
medication support for PCPs, and therapy was provided 
through what was essentially a specialty mental health 
clinic focused on primary care concerns. Convergence of 
perspectives at Site Zulu was much stronger than for 
other sites; all key informants agreed on the basic refer- 
ral process. Unique information was presented by each 
key informant, with each providing information specific 
to their domain of patient care. 

Variation between sites 

Key informants across the four sites described imple- 
mentation resistance and space limitations as common 
barriers to PC/MHI implementation, although this is not 
particularly informative for understanding how to facili- 
tate the intervention. A cursory review of these themes 
might suggest a common theme that coordination, col- 
laboration, and communication are important factors in 
PC/MHI implementation when sites choose to co-locate 
therapists in the primary care setting. However, analysis 
across sites revealed a between-site implementation 
theme that is more informative regarding the PC/MHI 
implementation. Between sites, the variation suggests 
that the preexisting structure of mental health care 
influenced how PC/MHI was implemented at each site. 
Further, communication and collaboration among both 
leaders and providers was critical to overcoming structural 



barriers that arose during PC/MHI implementation. The 
empirical support for both the within-site and between- 
site themes is presented in Table 5. 

PC/MHI implementation was characterized by conflicts 
between primary care and mental health, but communica- 
tion and collaboration between primary care and mental 
health were found to facilitate the implementation. As one 
example, psychiatrist resistance to the intervention was 
reported by primary care and the co-located mental health 
clinicians at both Site Alpha and Site Bravo. This resist- 
ance appeared to be due in part to how patient care was 
divided between primary care and mental health. Site 
Alpha faced challenges adjusting the formal agreements 
between the services, and Site Bravo encountered similar 
resistance in the physician suggestions to involve mental 
health in the care of medical illness. A key difference is 
that the Site Bravo MHL appeared to be responsive to 
PCP concerns, as negotiations were reported to result in 
tailored changes to the intervention implementation. In 
contrast, Site Alpha was characterized by one-sided plan- 
ning and poor communication between services, which 
limited collaborative care. Site Yankee reported a similar 
pattern of implementation as Site Alpha. These two sites 
are connected only loosely by their shared administrative 
structure. Thus, the one-sided planning at Site Yankee 
suggests that this style of implementation may have been 
common in the PC/MHI intervention and appears to have 
caused some potentially preventable implementation 
problems. 



Table 5 Empirical support for within-site and between-site implementation themes 



Site Within-site theme 

Alpha Coordination processes between services 
rather than structural factors required for 
implementation 



Bravo Communication and collaboration facilitated 
PC/MHI implementation 



Yankee Poor collaboration between primary care and 
mental health caused implementation 
problems 



Zulu Prior failure implementing co-located care 
influenced decision to physically separate 
services 



Support for within-site theme 

• Co-located structure present, but limited cross- 
service collaboration 

• POD and PC/MHI viewed as separate 
mechanisms for mental health access 

• Local tailoring of processes addressed structural 
space barrier 

• Inter-service collaboration resolved differences 



■ Open communication facilitates patient access 
and process improvement 

• Space conflict 

• ER referral procedure conflict 

• Different definitions of intervention success 

■ MH leader report of prior failure 

■ No divergent perspectives of shared 
phenomena, possibly due to physical separation 
between PC and MH 



Support for between-site theme 

• Space and preexisting consultation- 
liaison agreements are structural barriers 

• Neither barrier resolved by 
collaboration; space resolved through 
PCMH innovation 



■ Mutual awareness of concerns between 
services 

• Similar barriers to Site Alpha, but in Site 
Bravo, barriers were resolved through 
negotiation 

• Space and ER coverage agreements are 
structural barriers 

• Lack of collaboration appears to lead to 
conflicts over structural differences 

• Prior space limitation influenced 
implementation 

• PC/MHI adapted to preexisting space 
barrier 



Note: Between-site implementation theme: pre-existing structure influenced how PC/MHI was implemented; collaboration and cooperation among leaders and 
providers helped overcome these structural implementation barriers. POD = Psychiatrist on duty; PC/MHI = Primary Care/Mental Health Integration; ER = 
emergency room; PC = primary care; MH = mental health. 
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Available space was a structural barrier to PC/MHI im- 
plementation. Leadership responses to these barriers at 
Sites Yankee and Zulu may have exacerbated implementa- 
tion problems. Yankee leaders appear to have forced the 
implementation despite these constraints, whereas Zulu 
leaders chose to only co-locate psychiatrists to provide 
medication support to primary care and otherwise 
retained their model of referring patients to a geographic- 
ally separated annex for therapy. In contrast, at Site Alpha 
a psychologist developed group screening procedures to 
meet the high demand for mental healthcare. Referral 
processes were also structural barriers to PC/MHI im- 
plementation at Site Alpha (i.e., POD referrals), Bravo 
{i.e., degree to which PC/MHI should be concerned 
with medical issues), and Site Yankee (i.e., ER referrals). 
Differences between primary care and mental health 
were reported to have been resolved through collaboration 
among leaders at Site Bravo, and leaders resolved the ER 
conflict at Site Yankee. Thus, managing preexisting struc- 
tural conditions in Sites Alpha, Yankee, and Zulu was a 
key difference between sites that appeared to affect how 
PC/MHI was implemented. 

Discussion 

The current study demonstrates the value of analyzing 
both between-site and within-site variation among 
sources in qualitative implementation research. Analysis 
of variation across key informants facilitated the identifi- 
cation of between-site themes that the preexisting struc- 
ture of mental healthcare influenced how PC/MHI was 
implemented at each site and that collaboration among 
both leaders and providers were critical to overcoming 
structural barriers. Results suggest that if interviews had 
only been conducted with MHLs, then those perceptions 
would describe the broad characteristics of the interven- 
tion, but those interviews would provide less detail 
regarding the specific processes implemented, or the 
barriers to that implementation. Interviews with 
frontline staff revealed hidden conflicts and differences 
in interpretations of the same intervention. The analysis 
of qualitative data from multiple perspectives complicates 
and enriches studies of multisite intervention implementa- 
tion and contributes to the implementation science litera- 
ture in two ways. First, the examples of discrepancies 
between primary care and mental health provide a con- 
crete illustration of the practical implications of identify- 
ing both within-site and between-site implementation 
themes. Second, results have methodological implications 
for conceptualizing multisite implementation science 
research. 

Implications for implementation science 

Conclusions regarding the PC/MHI implementation can 
vary depending on whether data are analyzed within a 



site or across sites. Analyzing variation both within and 
across sites may reveal generalizable patterns of vari- 
ation. For example, each implementation was shown to 
be sensitive to the degree to which the intervention 
focused on issues related to medical conditions and the 
processes used to provide same-day urgent mental 
health access. Analyses within and across sites may also 
reveal variation in how the common barriers impacted 
implementations. For example, space was a barrier in all 
four sites, but the impact of space on the implementa- 
tion varied across sites, and perceived implications of 
space constraints tended to vary within site across the 
key informants. Finally, analysis across sites may high- 
light some important site-specific issues with PC/MHI 
implementation that may or may not generalize to other 
sites. For example, the distinction between a provider- 
centered and patient-centered perspective of interven- 
tion success was only revealed at Site Yankee, but the 
differences between informant perspectives may have 
occurred at other sites. Certainly, it is reasonable to 
expect that informants who are directly involved in an 
implementation are likely to view changes in processes 
as a success, whereas those who are outside the inter- 
vention may be more likely to expect tangible impacts 
on patient care, but additional data collection is needed 
to confirm this difference. 

Results indicated that administrative connections be- 
tween medical centers and community-based clinics 
accounted for minimal variation in implementation. Al- 
though there were some similarities between administra- 
tively connected sites (e.g., the practice of using lunch 
meetings to increase PCP buy-in was adopted at both 
Site Alpha and Site Bravo), results indicate that 
researchers should not assume that interventions are 
implemented the same in administratively connected 
sites. For example, in Site Alpha, the POD was 
considered outside the scope of the PC/MHI interven- 
tion, but the POD procedures were reported to affect 
the ability of PCMH providers to consult with specialty 
mental health providers. The POD procedures were a 
key feature of the organizational context for the inter- 
vention at this site. In contrast, psychiatrist resistance at 
Site Bravo was reported to be reduced through inter- 
service negotiation. The variation between these two 
administratively interconnected sites is likely due in part 
to differences in organization and mission between 
hospital-based and community-based outpatient clinics, 
but more broadly, history and preexisting processes used 
to provide mental healthcare at each site appeared to 
cause variation in the implementation barriers. As an- 
other example, both Site Zulu and their parent facility 
Site Yankee had previously implemented co-located care, 
but space limitations led to abandonment of that model. 
Site Yankee chose to attempt to reintegrate mental 
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health staff into the primary care setting, whereas Site 
Zulu chose instead to capitalize on the preexisting refer- 
ral processes. For medication support, Site Zulu provides 
limited access to specialty care psychiatrists already 
located in the same building. For therapy, Site Zulu 
continues to refer patients to a distant annex. Given the 
implementation difficulties at other sites, building on 
existing processes rather than attempting process 
redesign could be an effective strategy for improving 
patient access to mental healthcare. 

The current study also highlights an important issue 
when considering the fit of an intervention with a spe- 
cific organizational context. Implementation science has 
long recognized that misfit can be solved by either modi- 
fying existing practices or modifying interventions [46]. 
Three of the four sites chose to modify existing 
practices, but Site Zulu chose instead to modify the 
intervention by providing limited co-located mental 
healthcare through medication support and use the 
intervention funding to expand their existing mental 
health services. Given that space restrictions led to prior 
efforts at Site Zulu to provide co-located care to fail, this 
choice is certainly rational. Indeed, the goals of the PC/ 
MHI intervention {e.g., improving access to mental 
health services) may be more easily achieved by working 
within the parameters of existing procedures rather than 
attempting large-scale organizational changes that con- 
flict with local operating conditions (e.g., implementing 
co-located care in a site without the necessary space). 
However, the benefits and tradeoffs in this type of inter- 
vention modification are not clear in the current study. 
This may be a fruitful area for future research to help 
plan more effective inter-service interventions. 

Methodological implications 

The key informant interview is a ubiquitous method- 
ology in implementation science, but sampling for 
multisite qualitative implementation studies can be more 
complex than in other qualitative health research. Quali- 
tative theoretical sampling aimed at revealing grounded 
theory is typically conducted until thematic saturation is 
reached, that is, until no new information is discovered 
[44]. However, the current study indicates that sampling 
can be complex in multisite implementation studies with 
multidisciplinary workgroups because variation may 
occur across professions and positions within site and 
between sites. Thus, the meaning of saturation may 
change depending on the populations being considered. 
Implementation study designs should consider the 
implications of these variations for generalizability of 
results and validity of conclusions. 

Conceptual frameworks guiding qualitative implemen- 
tation research should be explicit regarding variation 
among informants so readers can gauge initial 



standpoints and assumptions of the researchers (e.g., the 
degree to which researchers allowed for different types 
of converging/diverging perceptions to surface). Building 
implementation theories on convergent perceptions can 
limit the practical utility of those theories in accounting for 
divergent factors, such as differences in perceived responsi- 
bilities and unintended consequences of interventions. The 
current study provides concrete examples of how and why 
key informants may perceive different aspects of an inter- 
vention. For example, convergent perceptions from mental 
health focused on how the organization structure affected 
processes (e.g., co-location promotes positive interactions). 
In contrast, both PCMH and PCP were able to provide 
detail regarding the specific processes used to coordin- 
ate patient care. PCMH identified specific structural 
barriers to the implementation (e.g., limited implemen- 
tation of co-location, lack of education in nursing and 
clerical staff). PCP reported unique details regarding 
the implemented practices and outcomes (e.g., can 
knock on PCMH door if needed, patient referral com- 
pliance has improved). Without these multiple frontline 
perspectives at each site, neither the within-site nor the 
between-site implementation themes could have been 
identified. 

It is important that implementation researchers under- 
stand their role in processing and presenting divergent 
perceptions [43]. Divergence can be as important as con- 
vergence in understanding a phenomenon, but the 
uniqueness of divergent perceptions may cause some to 
question their validity [42]. Multisite studies may present 
specific concerns regarding the credibility of divergent 
perceptions because within-site sampling is likely to be 
restricted in order to sample across sites. However, 
multisite studies may have a unique advantage in the 
analysis of divergent perceptions. The current study 
established the credibility of divergent perceptions by 
identifying consistent patterns of distorted perceptions 
across the multidisciplinary workgroups within a specific 
source [45]. 

We propose that distortions in PC/MHI-related 
perceptions were derived from differences in key 
informants' work roles. For example, the PCP and MHL 
disagreed regarding the key barriers to implementation 
in Site Alpha. PCP described the lack of collaboration 
from psychiatrists, whereas PCMH reported that getting 
buy-in from PCPs was the biggest challenge. As another 
example at Site Yankee, the PCP and MHL disagreed 
regarding the definition of implementation success. 
PCMH believed that intervention success was indicated 
by increased referrals from PCPs, whereas the PCP 
believed that intervention success was indicated by 
increased patient compliance. Conclusions based on 
interviews with only limited types of frontline staff in 
multidisciplinary work environments may be biased 
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toward the perspective of those positions (or other rele- 
vant characteristics). Thus, while multisite implementa- 
tion studies can be conducted with a limited sample of 
organizational positions, the validity of the conclusions 
based on the data may be limited. Implementation 
researchers should consider the benefits and tradeoffs of 
cost-effectiveness versus comprehensiveness regarding 
collecting data from multiple positions in their studies. 
The current study indicates that complete coverage of 
the key multidisciplinary positions impacted by the 
intervention (e.g., primary care and co-located mental 
health frontline staff) is needed to minimize bias in 
conclusions made by researchers regarding the success 
of implementation. 

Methodologically, the current study affirms the value 
of concurrent data collection and analysis during imple- 
mentation research [47]. Perceptions from one source 
can be compared with other sources, and divergent 
perceptions should be carefully probed during data 
collection to understand why these factors are not 
shared among informants, whether there are systematic 
patterns of divergence, and/ or why informants would fail 
to report the divergent factors. For example, the unilat- 
eral intervention planning by mental health was noted 
as a barrier at Sites Alpha, Bravo, and Yankee, but was 
only revealed in front-line interviews. Site Alpha only 
had one co-located psychologist. If this report was not 
corroborated by the primary care interviews, then cred- 
ibility would be questionable in a single-site study. How- 
ever, the pattern of responses across sites suggests that 
unilateral planning by mental health was a common 
practice in PC/MHI implementation. In this example, 
further data collection could have been conducted at 
Site Yankee and the other sites to understand why MHLs 
did or did not involve PCPs in their implementation 
planning. 

In the qualitative study design, researchers should build 
in time for discussion and reflection on convergent and 
divergent perceptions between interviews. This would 
provide opportunities for implementation researchers to 
revise sampling strategies and interview questions as 
seemingly relevant patterns of divergence emerge. Al- 
though these types of designs require significant commit- 
ment during the data collection process, we suggest that 
they are cost effective in that these designs may greatly 
enhance the effectiveness of implementation research. 

Limitations 

There were three main limitations to the current study. 
First, as the data collection was part of a multisite evalu- 
ation, the number of informants at a single site was neces- 
sarily limited. The current study did not analyze variation 
within employee positions. For example, the sampling was 
designed to identify the most knowledgeable informants, 



but interviewing less engaged primary care staff could 
have been valuable to further probe discrepancies between 
primary care and mental health. It is possible that within- 
informant differences could impact perceptions of the 
intervention, and potentially impact the utility of 
conclusions. Second, these sites were purposefully sampled 
from the evaluation sites because initial data analysis 
indicated both convergence and divergence. These are four 
case studies and may not be typical of PC/MHI interven- 
tion implementations in the VA. The sites were selected to 
provide an illustrative example of the importance of 
reporting both convergent and divergent perceptions in 
implementation studies. Third, this study was based on 
archival data. There was no opportunity to collect add- 
itional data to clarify divergent perceptions, but we 
acknowledge that additional data collection would have 
been valuable in exploring the discrepancies identified in 
the current study. It is our experience that this type of 
design is common in the practice of intervention evalu- 
ation. Studying perceptual divergence may require modifi- 
cation to study designs to promote concurrent analysis. 
Rapid analysis after data collection can identify divergent 
perceptions that may be probed in follow-up interviews. 
This type of design is common in longitudinal qualitative 
methods, such as ethnographies, but may also be critically 
important in implementation research. Evaluation designs 
that include planned follow-up interviews should be 
promoted by both researchers and funding agencies, but if 
funding does not allow for follow-up interviews, at the very 
least, implementation researchers should be sure to collect 
perceptions from all of the relevant stakeholders who are 
impacted by an intervention. 

Conclusion 

The current study demonstrated that attention toward 
variation across organizational positions and sites can 
enrich implementation research by illustrating how 
shared themes are perceived by different stakeholders. 
This variation may be important when replicating 
interventions as a deep understanding of implementa- 
tion factors may be needed to align interventions with 
varying stakeholder needs. Divergent perceptions were 
found to be potentially important information for 
understanding why an implementation may or may not 
succeed. Given the time and funding limitations inher- 
ent in implementation research, it is important to 
emphasize the benefits and design implications inher- 
ent in analyzing divergent perceptions in order to 
increase the quality and effectiveness of implementa- 
tion studies. 

Endnote 

a Triangulation can be used to resolve variation among 
sources, methods, investigators, or theories, but this 
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paper is concerned with the most common use, triangu- 
lation among sources. 
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